Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

processor: Implement native processor metrics_selector #8526

Merged
merged 1 commit into from
Mar 14, 2024
Merged

processor: Implement native processor metrics_selector #8526

merged 1 commit into from
Mar 14, 2024

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Feb 27, 2024 •
edited
Loading

For processing metrics, we need to move using processors because there is less motivations in the current implementation to use processors based pipeline.
If we provide filtering feature on this mechanism, users want to use processors from the classic configuration.

Also, ordinary filters are intended to use non-processor world due to msgpack based ingestion on ingesting filters. This is really hard to distinguish which types of events: logs, metrics, and traces.
Instead, the interface of processor provides easily to be able to distinguish three types of events.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
service:
  flush: 5
  daemon: off
  log_level:  debug

pipeline:
  inputs:
    - name: fluentbit_metrics
      tag: fluentbit.metrics
      scrape_interval: 10

      processors:
        metrics:
          - name: metrics_selector
            metric_name: /storage/
            action: include
          - name: metrics_selector
            metric_name: /fs/
            action: exclude

          - name: labels
            delete: name


  outputs:
    - name: stdout
      match: '*'
  • Debug log output from testing the change
Fluent Bit v3.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

___________.__                        __    __________.__  __          ________  
\_   _____/|  |  __ __   ____   _____/  |_  \______   \__|/  |_  ___  _\_____  \ 
 |    __)  |  | |  |  \_/ __ \ /    \   __\  |    |  _/  \   __\ \  \/ / _(__  < 
 |     \   |  |_|  |  /\  ___/|   |  \  |    |    |   \  ||  |    \   / /       \
 \___  /   |____/____/  \___  >___|  /__|    |______  /__||__|     \_/ /______  /
     \/                     \/     \/               \/                        \/ 

[2024/02/27 14:49:56] [ info] Configuration:
[2024/02/27 14:49:56] [ info]  flush time     | 5.000000 seconds
[2024/02/27 14:49:56] [ info]  grace          | 5 seconds
[2024/02/27 14:49:56] [ info]  daemon         | 0
[2024/02/27 14:49:56] [ info] ___________
[2024/02/27 14:49:56] [ info]  inputs:
[2024/02/27 14:49:56] [ info]      fluentbit_metrics
[2024/02/27 14:49:56] [ info] ___________
[2024/02/27 14:49:56] [ info]  filters:
[2024/02/27 14:49:56] [ info] ___________
[2024/02/27 14:49:56] [ info]  outputs:
[2024/02/27 14:49:56] [ info]      stdout.0
[2024/02/27 14:49:56] [ info] ___________
[2024/02/27 14:49:56] [ info]  collectors:
[2024/02/27 14:49:56] [ info] [fluent bit] version=3.0.0, commit=f9ed76af1b, pid=997805
[2024/02/27 14:49:56] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/02/27 14:49:56] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/02/27 14:49:56] [ info] [cmetrics] version=0.7.0
[2024/02/27 14:49:56] [ info] [ctraces ] version=0.4.0
[2024/02/27 14:49:56] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] initializing
[2024/02/27 14:49:56] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] storage_strategy='memory' (memory only)
[2024/02/27 14:49:56] [ info] [output:stdout:stdout.0] worker #0 started
[2024/02/27 14:49:56] [debug] [fluentbit_metrics:fluentbit_metrics.0] created event channels: read=21 write=22
[2024/02/27 14:49:56] [ info] [processor:selector:selector.0] OR mode
[2024/02/27 14:49:56] [debug] [stdout:stdout.0] created event channels: read=23 write=24
[2024/02/27 14:49:56] [ info] [sp] stream processor started
[2024/02/27 14:50:06] [debug] [input chunk] update output instances with new chunk size diff=3314, records=0, input=fluentbit_metrics.0
[2024/02/27 14:50:11] [debug] [task] created task=0x6195f70 id=0 OK
2024-02-27T05:50:06.314435517Z fluentbit_uptime{hostname="cosmo-desktop2"} = 10
2024-02-27T05:49:56.491818462Z fluentbit_input_bytes_total = 0
2024-02-27T05:49:56.491818462Z fluentbit_input_records_total = 0
2024-02-27T05:50:06.310958339Z fluentbit_input_metrics_scrapes_total = 1
2024-02-27T05:49:56.516306480Z fluentbit_output_proc_records_total = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_proc_bytes_total = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_errors_total = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_retries_total = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_retries_failed_total = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_dropped_records_total = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_retried_records_total = 0
2024-02-27T05:50:06.314435517Z fluentbit_process_start_time_seconds{hostname="cosmo-desktop2"} = 1709012996
2024-02-27T05:50:06.314435517Z fluentbit_build_info{hostname="cosmo-desktop2",version="3.0.0",os="linux"} = 1709012996
2024-02-27T05:50:06.314435517Z fluentbit_hot_reloaded_times{hostname="cosmo-desktop2"} = 0
2024-02-27T05:49:56.491818462Z fluentbit_input_ingestion_paused = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_upstream_total_connections = 0
[2024/02/27 14:50:11] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-02-27T05:49:56.516306480Z fluentbit_output_upstream_busy_connections = 0
2024-02-27T05:49:56.516306480Z fluentbit_output_chunk_available_capacity_percent = 100
[2024/02/27 14:50:11] [debug] [out flush] cb_destroy coro_id=0
[2024/02/27 14:50:11] [debug] [task] destroy task=0x6195f70 (task_id=0)
^C[2024/02/27 14:50:12] [engine] caught signal (SIGINT)
[2024/02/27 14:50:12] [ warn] [engine] service will shutdown in max 5 seconds
[2024/02/27 14:50:12] [ info] [input] pausing fluentbit_metrics.0
[2024/02/27 14:50:12] [ info] [engine] service has stopped (0 pending tasks)
[2024/02/27 14:50:12] [ info] [input] pausing fluentbit_metrics.0
[2024/02/27 14:50:12] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/02/27 14:50:12] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==997805== 
==997805== HEAP SUMMARY:
==997805==     in use at exit: 0 bytes in 0 blocks
==997805==   total heap usage: 6,236 allocs, 6,236 frees, 1,330,806 bytes allocated
==997805== 
==997805== All heap blocks were freed -- no leaks are possible
==997805== 
==997805== For lists of detected and suppressed errors, rerun with: -s
==997805== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920 cosmo0920 force-pushed the cosmo0920-processor-selector branch from 8287e9d to 6f770a7 Compare February 27, 2024 06:21
@cosmo0920 cosmo0920 force-pushed the cosmo0920-processor-selector branch from 8c5a365 to a68aec7 Compare February 27, 2024 07:29
patrick-stephens
patrick-stephens reviewed Feb 27, 2024
View reviewed changes
Copy link
Contributor

@patrick-stephens patrick-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So @cosmo0920 is the intended effect here that we can do some exclusion or inclusion of specific values for the processor (as matching does not function for processors)? We can say "run/ignore this processor for this record" basically?

We should update the docs as well for this: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file#config_processors

I would like us to add processors to the smoke tests as well, it's a good example plus picks up failures if we break YAML config: https://github.com/fluent/fluent-bit/blob/master/packaging/testing/smoke/container/fluent-bit.yaml

CMakeLists.txt Outdated
@@ -284,6 +284,7 @@ option(FLB_FILTER_NIGHTFALL "Enable Nightfall filter"
option(FLB_FILTER_WASM "Enable WASM filter" Yes)
option(FLB_PROCESSOR_LABELS "Enable metrics label manipulation processor" Yes)
option(FLB_PROCESSOR_ATTRIBUTES "Enable atributes manipulation processor" Yes)
option(FLB_PROCESSOR_SELECTOR "Enable selector processor" Yes)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would invert the description here:

Suggested change
option(FLB_PROCESSOR_SELECTOR "Enable selector processor" Yes)
option(FLB_PROCESSOR_SELECTOR "Enable processor extended selector" Yes)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to use ambiguous description because we're planning to extend a capability to handle traces in the future.
Or, how about using "Enable selector for non-logs type of events" or something similar?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This processor is a non-log types parity of logs type handling of filter_grep.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just make it a processor_grep then? That makes more sense to me, it feels like we're just working around the lack of grep support for metrics and traces.

So this won't work for logs then, so it's not really a processor itself but a specific type of metrics (only?) grep processor?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is because normal processor_grep will be collided for name collisions. Because normal filters are able to use as processors.

@cosmo0920
Copy link
Contributor Author

So @cosmo0920 is the intended effect here that we can do some exclusion or inclusion of specific values for the processor (as matching does not function for processors)? We can say "run/ignore this processor for this record" basically?

Currently, we're implementing this for taking exclusion effects from the metrics type of events.

@cosmo0920 cosmo0920 force-pushed the cosmo0920-processor-selector branch from 38e6d26 to 5c02d83 Compare March 6, 2024 11:11
@patrick-stephens patrick-stephens mentioned this pull request Mar 6, 2024
Closed
@cosmo0920 cosmo0920 changed the title processor: Implement native processor selector processor: Implement native processor metrics_selector Mar 13, 2024
@cosmo0920 cosmo0920 force-pushed the cosmo0920-processor-selector branch from d8beec7 to 38b5893 Compare March 14, 2024 05:26
@cosmo0920 cosmo0920 force-pushed the cosmo0920-processor-selector branch from 38b5893 to 53979d3 Compare March 14, 2024 05:28
@cosmo0920
processor_metrics_selector: Implement native processor for metrics ma…
96d4cc1 
…nipulations

* Prevent dangling pointer if cmt context is recreated on input_metric and processor.
* processor_labels: Follow change of the signature of metrics callback
* lib: Support setter for processor to make testing processors easily

Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920 cosmo0920 force-pushed the cosmo0920-processor-selector branch from 53979d3 to 96d4cc1 Compare March 14, 2024 05:29
@edsiper edsiper merged commit a7b3d9b into master Mar 14, 2024
43 of 44 checks passed
@edsiper edsiper deleted the cosmo0920-processor-selector branch March 14, 2024 21:43
@BrewTestBot BrewTestBot mentioned this pull request Mar 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrick-stephens patrick-stephens patrick-stephens left review comments

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@niedbalski niedbalski Awaiting requested review from niedbalski niedbalski is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@celalettin1286 celalettin1286 Awaiting requested review from celalettin1286 celalettin1286 is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cosmo0920 @edsiper @patrick-stephens